Player biofeedback for dynamically controlling a video game state

ABSTRACT

Various embodiments are directed towards employing one or more physical sensors arranged on or in proximity to a video game player to obtain biofeedback measures that are then usable to dynamically modify a state of play of a video game. The sensors may be connected or even unconnected to the game player, replace, or otherwise augment traditional physical game controllers. The sensors gather various biofeedback measures and provide such measures to a biofeedback application programming interface (API). Before and/or during video game play, the video game queries the biofeedback API to request inferences about the game player&#39;s state of arousal. The response to the query is then used to modify the state of the video game play. Where the video game is a multi-player video game, biofeedback measures from other game players may also be obtained and used to further modify the state of the video game play.

TECHNICAL FIELD

The present invention relates generally to interactive video games andmore particularly, but not exclusively, to using biofeedback from a gameplayer to modify a video game state.

BACKGROUND

Today, the computer gaming industry is a multi-billion dollar industry.Such popularity may be due in part to faster computing devices, higherquality graphics, and better quality games. Many of today's video gamesprovide a variety of different input/output devices usable by a gameplayer to interact with the game. For example, many video games allow aplayer to interact using a keyboard and/or mouse. While suchinput/output controllers allow the game player to interact with thegame, the game player might not ‘feel’ that they are immersed into thegame. Therefore, many video games have been redesigned to allow use ofgamepads, joysticks, trackballs, game paddles, and the like to provideways of immersing the video game player. Some joysticks and/or paddlesare configured to resemble a type of device consistent with the videogame being played. For example, for some flight simulation games, ajoystick might be designed to provide throttle quadrants, levels,wheels, and handheld sticks that appear to the game player as thoughthey are flying within a cockpit of an aircraft.

By modifying the input devices, video game players are more likely tobecome involved with and therefore, enjoy the video game more so. Assuch, the video game player is more likely to continue to play the game,share the game with others, and perhaps to purchase similar games in thefuture. This trend of modifying the input devices to have more gameplayer involvement is even more apparent with the advent of wirelesscontrollers. For example, in one popular video game, the game inputcontroller is a wireless handheld controller that may include built-inaccelerometers, infrared detectors, or similar components. Suchcomponents are used to sense a position of the controller inthree-dimensional space when pointed at a light emitting diode (LED)within a remote sensor bar. The game player then controls the game usingphysical gestures as well as traditional buttons, to play games such asbowling, imaginary musical instruments, boxing games, or the like.

However, while many game players may feel this provides an increasedlevel of involvement in the video game, other game players may stillfeel that the involvement in the video game is incomplete. Thus, it iswith respect to these considerations, and others, that the presentinvention has been made.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present invention aredescribed with reference to the following drawings. In the drawings,like reference numerals refer to like parts throughout the variousfigures unless otherwise specified.

For a better understanding of the present invention, reference will bemade to the following Detailed Description of the Invention, which is tobe read in association with the accompanying drawings wherein:

FIG. 1 shows a functional block diagram illustrating one embodiment ofan environment for practicing the invention;

FIG. 2 shows one embodiment of a client device for use in theenvironment of FIG. 1;

FIG. 3 shows one embodiment of a network device for use in theenvironment of FIG. 1;

FIG. 4 illustrates a flow chart for one embodiment of a process ofemploying biofeedback measurements from a game player to modify a gameplay state in a video game;

FIG. 5 illustrates a flow chart for one embodiment of a process ofperforming an analysis of biofeedback measures from a game player foruse in the video game;

FIG. 6 illustrates one embodiment of a non-exhaustive, non-limitingexample of queries for use in querying a biofeedback applicationprogramming interface (API) for biofeedback measures;

FIG. 7 illustrates one embodiment of a non-exhaustive non-limitingexample of using biofeedback measures for use in modifying a game playstate in an arena combat video game; and

FIG. 8 illustrates one embodiment of a non-exhaustive non-limitingexample of using biofeedback measures for use in modifying a game playstate in a space video game.

DETAILED DESCRIPTION

The present invention now will be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific exemplary embodiments bywhich the invention may be practiced. This invention may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein; rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of the invention to those skilled in the art.Among other things, the present invention may be embodied as methods ordevices. Accordingly, the present invention may take the form of anentirely hardware embodiment, an entirely software embodiment or anembodiment combining software and hardware aspects. The followingdetailed description is, therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of “in” includes “in” and “on.”

As used herein, the terms “biofeedback,” and “physiological” refer tomeasures of a game player's specific and quantifiable bodily functions.Such biofeedback measures are typically also referred to as measurementsof unconscious or involuntary bodily functions. Such biofeedbackmeasures may include, but are not limited to blood pressure, heartrates, eye movements, pupil dilations, skin temperatures, sweat glandactivity, muscle tensions, and similar bodily functions. As describedfurther herein, such measures are usable to make inferences about thegame player's state of arousal or emotional state. It is noted that astate of arousal includes not only an emotional state, but aphysiological state as well. Moreover, as used herein, a state ofarousal further includes determination of engagement, valence, and/orother user states based on physiological measurements.

The following briefly describes the embodiments of the invention inorder to provide a basic understanding of some aspects of the invention.This brief description is not intended as an extensive overview. It isnot intended to identify key or critical elements, or to delineate orotherwise narrow the scope. Its purpose is merely to present someconcepts in a simplified form as a prelude to the more detaileddescription that is presented later.

Briefly stated, various embodiments are directed towards employing oneor more physical sensors arranged on or in proximity to a video gameplayer to obtain biofeedback measures about the game player that isusable to dynamically modify a state of play of the video game. In oneembodiment, the modifications may be performed substantially inreal-time. In another embodiment, the modifications may be performed foruse in a subsequent game play. The physical sensors may be connected tothe game player, replace, and/or otherwise augment traditional physicalgame controllers. In another embodiment, the physical sensors need notbe connected to the game player and may instead be located in proximityto the game player. Non-limiting examples of such physically unconnectedsensors include a video camera, weight/position sensor pads upon whichthe game player might stand upon, or the like. The sensors are arrangedto gather various biofeedback measures such as heart activity, galvanicskin responses, body temperatures, eye movements, head or other bodymovements, or the like, and to provide such measures to a biofeedbackapplication programming interface (API). Before and/or during a videogame play, the video game may query the biofeedback API for an inferenceabout the game player's state of arousal, emotional state, or the like,as described further below based on the biofeedback measures. Then,based on the response to the query, the video game modifies a state ofvideo game play. In this manner, the video game may determine whetherthe game player's current physiological state is consistent with a typeand/or level of experience the video game may seek to provide. Forexample, if the game player's stress or arousal state is determined tobe above a given threshold, the video game may modify the state of thegame play to provide the game player an opportunity to relax and/orrecover. In another embodiment, where the game player's stress orarousal state is determined to be below another threshold, the videogame may modify the state of the game play to provide an increased levelof excitement for the game player.

In one embodiment, the threshold may be based on historical biofeedbackmeasures and/or inferences about the particular game player. In anotherembodiment, the threshold may be based on analysis of the particulargame player for the current video game play. In still anotherembodiment, the threshold may be based on statistical analysis of aplurality of game players.

In one embodiment, where the video game is configured as a multi-playervideo game, biofeedback measures from other game players may also beobtained and used to further modify a state of the video game play.

Illustrative Operating Environment

FIG. 1 illustrates a block diagram generally showing an overview of oneembodiment of a system in which the present invention may be practiced.System 100 may include many more components than those shown in FIG. 1.However, the components shown are sufficient to disclose an illustrativeembodiment for practicing the present invention. As shown in the figure,system 100 includes local area networks (“LANs”)/wide area networks(“WANs”)−(network) 105, wireless network 110, client device 101, gameserver device (GSD) 110, and biofeedback sensors 120.

One embodiment of a client device usable as client device 101 isdescribed in more detail below in conjunction with FIG. 2. Briefly,however, client device 101 may include virtually any mobile computingdevice capable of receiving and sending a message over a network, suchas network 110, or the like. Such devices include portable devices suchas, radio frequency (RF) devices, infrared (IR) devices, PersonalDigital Assistants (PDAs), game consoles, handheld computers, laptopcomputers, wearable computers, tablet computers, integrated devicescombining one or more of the preceding devices, or the like. Clientdevice 101 may also include virtually any computing device thattypically connects using a wired communications medium, such as network105, such as personal computers, multiprocessor systems,microprocessor-based or programmable consumer electronics, network PCs,or the like. Thus, in one embodiment, client device 101 may beconfigured to operate over a wired and/or a wireless network.

Client device 101 typically range widely in terms of capabilities andfeatures. For example, a handheld device may have a numeric keypad and afew lines of monochrome LCD display on which only text may be displayed.In another example, a web-enabled client device may have a touchsensitive screen, a stylus, and several lines of color LCD display inwhich both text and graphics may be displayed.

A web-enabled client device may include a browser application that isconfigured to receive and to send web pages, web-based messages, or thelike. The browser application may be configured to receive and displaygraphics, text, multimedia, or the like, employing virtually any webbased language, including a wireless application protocol messages(WAP), or the like. In one embodiment, the browser application isenabled to employ Handheld Device Markup Language (HDML), WirelessMarkup Language (WML), WMLScript, JavaScript, Standard GeneralizedMarkup Language (SMGL), HyperText Markup Language (HTML), eXtensibleMarkup Language (XML), or the like, to display and send information.

Client device 101 also may include at least one application that isconfigured to receive content from another computing device. Theapplication may include a capability to provide and receive textualcontent, multimedia information, components to a computer application,such as a video game, or the like. The application may further provideinformation that identifies itself, including a type, capability, name,or the like. In one embodiment, client device 101 may uniquely identifythemselves through any of a variety of mechanisms, including a phonenumber, Mobile Identification Number (MIN), an electronic serial number(ESN), mobile device identifier, network address, or other identifier.The identifier may be provided in a message, or the like, sent toanother computing device.

Client device 101 may also be configured to communicate a message, suchas through email, Short Message Service (SMS), Multimedia MessageService (MMS), instant messaging (IM), internet relay chat (IRC),Mardam-Bey's IRC (mIRC), Jabber, or the like, between another computingdevice. However, the present invention is not limited to these messageprotocols, and virtually any other message protocol may be employed.Thus, in one embodiment, client device 101 may enable users toparticipate in one or more messaging sessions, such as a chat session, agaming session with messaging, or the like. Such messaging sessions maybe text oriented, in that the communications are achieved using text.However, other messaging sessions may occur using client device 101 thatemploy other mechanisms to communicate, include, but not limited toaudio, graphics, video, and/or a combination of text, audio, graphics,and/or video.

Client device 101 may be configured to receive messages, images, and/orother biofeedback measures, from various biofeedback sensors 120.Illustrated in FIG. 1 are non-limiting, non-exhaustive examples ofpossible physical biofeedback sensors 120 that may be connected orunconnected to the user, replace, and/or otherwise augment traditionalphysical game controllers. Thus, as illustrated biofeedback sensors 120may be integrated within a game controller (sensor 123), one or morekeys, wheels, or the like, on a keyboard (sensor 124). In oneembodiment, the game controller may include modular and/or pluggablecomponents that may include modular and/or pluggable sensors (123).

Similarly, biofeedback sensors 120 may include a camera 121, a touch pad122, or even a head device 125. However, as noted, other biofeedbacksensors 120 may also be employed, including, eyeglasses, wrist bands,finger sensor attachments, sensors integrated within or on a computermice, microphones for measuring various voice patterns, or the like.Thus, it should be apparent to one skilled in the art that variousembodiments may employ virtually any mechanism configurable to obtainbiofeedback measures of the game player.

The biofeedback sensors 120 may be arranged to gather various measuresof a game player before, after, and/or during a video game play. Suchmeasures include, but are not limited to heart rate and/or heart ratevariability; galvanic skin responses; body temperature; eye movement;head, face, hand, or other body movement, gestures, positions, facialexpressions, postures, or the like. Additionally, biofeedback sensors120 may collect other measures, including, blood oxygen levels, otherforms of skin conductance levels, respiration rate, skin tension, voicestress levels, voice recognition, blood pressure, Electroencephalography(EEG) measures, Electromyography (EMG) measures, response times, or thelike.

Biofeedback sensors 120 may provide the measures to client device 101.In one embodiment, the measures may be provided to client device 101over any of a variety of wired and/or wireless connections. Thus,biofeedback measures may be communicated over various cables, wires, orthe like, with which other information may also be communicated for agame play. For example, biofeedback measures might be transmitted over aUSB cable, coaxial cable, or the like, with which a mouse, keyboard,game controller, or the like, is also coupled to client device 101.However, in another embodiment, a distinct wired connection may beemployed. Similarly, biofeedback sensors 120 may employ various wirelessconnections to communicate biofeedback measures. In addition, any of avariety of communication protocols may be used to communicate themeasures. Thus, the invention is not to be construed as being limited toa particular wired or wireless communication mechanism and/orcommunication protocol.

In one embodiment, client device 101 may include a biofeedback deviceinterface (BFI) that is configured to determine whether one or morephysical sensors 120 are operational, and to manage receipt ofbiofeedback measures from physical sensors 120. One embodiment of a BFIis described in more detail below in conjunction with FIG. 2. Briefly,however, the BFI may further timestamp the received biofeedbackmeasures, buffer at least some of the measures, and/or forward themeasures to GSD 110 for use in modifying a state of a video game play.Buffering of the received biofeedback measures may enable the BFI toperform quality analysis upon the received measures, and to providealert messages based on a result of the analysis.

Wireless network 110 is configured to couple client device 101 withnetwork 105. Wireless network 110 may include any of a variety ofwireless sub-networks that may further overlay stand-alone ad-hocnetworks, or the like, to provide an infrastructure-oriented connectionfor client device 101. Such sub-networks may include mesh networks,Wireless LAN (WLAN) networks, cellular networks, or the like.

Wireless network 110 may further include an autonomous system ofterminals, gateways, routers, or the like connected by wireless radiolinks, or the like. These connectors may be configured to move freelyand randomly and organize themselves arbitrarily, such that the topologyof wireless network 110 may change rapidly.

Wireless network 110 may further employ a plurality of accesstechnologies including 2nd (2G), 3rd (3G), 4th (4G) generation radioaccess for cellular systems, WLAN, Wireless Router (WR) mesh, or thelike. Access technologies such as 2G, 2.5G, 3G, 4G, and future accessnetworks may enable wide area coverage for client devices, such asclient device 101 with various degrees of mobility. For example,wireless network 110 may enable a radio connection through a radionetwork access such as Global System for Mobile communication (GSM),General Packet Radio Services (GPRS), Enhanced Data GSM Environment(EDGE), Wideband Code Division Multiple Access (WCDMA), Bluetooth, orthe like. In essence, wireless network 110 may include virtually anywireless communication mechanism by which information may travel betweenclient device 101 and another computing device, network, or the like.

Network 105 is configured to couple computing devices, such as GSD 110to other computing devices, including potentially through wirelessnetwork 110 to client device 101. However, as illustrated, client device101 may also be connected through network 105 to GSD 110. In any event,network 105 is enabled to employ any form of computer readable media forcommunicating information from one electronic device to another. Also,network 105 can include the Internet in addition to local area networks(LANs), wide area networks (WANs), direct connections, such as through auniversal serial bus (USB) port, other forms of computer-readable media,or any combination thereof. On an interconnected set of LANs, includingthose based on differing architectures and protocols, a router acts as alink between LANs, enabling messages to be sent from one to another.Also, communication links within LANs typically include twisted wirepair or coaxial cable, while communication links between networks mayutilize analog telephone lines, full or fractional dedicated digitallines including T1, T2, T3, and T4, Integrated Services Digital Networks(ISDNs), Digital Subscriber Lines (DSLs), wireless links includingsatellite links, or other communications links known to those skilled inthe art. Furthermore, remote computers and other related electronicdevices could be remotely connected to either LANs or WANs via a modemand temporary telephone link. In essence, network 105 includes anycommunication method by which information may travel between computingdevices.

One embodiment of GSD 110 is described in more detail below inconjunction with FIG. 3. Briefly, however, GSD 110 may include anycomputing device capable of connecting to network 105 to enable a userto participate in one or more online games, including, but not limitedmulti-player games, as well as single player games. Thus, while FIG. 1illustrates a single client device 101 with biofeedback sensors 120, theinvention is not so limited, and a plurality of similar client deviceswith biofeedback sensors may be deployed within system 100.

Therefore, GSD 110 is configured to receive various biofeedback measuresfrom one or more game players and to employ the received measures tomodify a state of the video game. GSD 110 may employ the biofeedback todynamically adjust a game play difficulty, and/or other aspects of avideo game based on the biofeedback measures. For example, in oneembodiment, if it is determined that a user is experiencing a level ofstress defined as excessive, based on a threshold, the video game withinGSD 110 might provide a different game play to enable reduction in thedetermined stress level.

GSD 110 may also enable the video game to provide a unique experienceeach time it is played based on the biofeedback measures of the gameplayer. For example, in one embodiment, a color of an object, size,shape, and/or action of a game character, or the like, may be adjustedbased on biofeedback measures. That is various aspects of a backgrounddisplayed within the background of the game may be modified based on theresults of an analysis of the biofeedback measures.

In one embodiment, historical measurements may be stored, and analyzedto enable GSD 110 to detect a particular game player. Such storedmeasurements may then be used to personalize the game play for theparticular game player, identify changes in a game play by theparticular game player based on a determined trend determination, or thelike. In one embodiment, historical measurements together with analysisof the biofeedback measures may be used to determine whether the gameplayer is currently associated with a prior user profile—that is, isthis game player someone that has played before? GSD 110 may also adjusta type of game play offered based a determination of the game player'slevel of engagement during a game play, historical patterns, or thelike.

GSD 110 may further provide matchmaking decisions based in whole or inpart on a physiological or emotional state of a game player that mayseek a multiplayer game session. In still other embodiments, GSD 110 maydynamically adjust game play instructions, tutorials, or the like, basedon the received biofeedback measures. For example, where it might bedetermined that the game player is determined to be bored or otherwiseuninterested in the instructions, tutorials, or the like, GSD 110 mightenable the material to be sped up, skipped or the like.

GSD 110 is not limited to these examples of how biofeedback measures maybe used however, and others ways of employing the biofeedback measuresto modify a game play state may also be used. For example, thebiofeedback measures may be employed to directly control an aspect ofthe game play. One non-limiting example of such is described in moredetail below in conjunction with FIG. 8.

In still other embodiments, GSD 110 may depict the game player'semotional, physiological state and/or other aspects of the game player'sexpression within a game character. For example, a game player's avatarmight be modified to display a heart that beats at the rate of the gameplayer's heart, or the avatar might be shown to breathe at the gameplayer's rate, or sweat, or even show a facial expression, or bodyposition based on the received biofeedback measures for the game player.Thus, GSD 110 may employ biofeedback measures in any of a variety ofways to modify a state of a game play.

Devices that may operate as GSD 110 include personal computers, desktopcomputers, multiprocessor systems, video game consoles,microprocessor-based or programmable consumer electronics, network PCs,server devices, and the like.

Moreover, although GSD 110 is illustrated as a single network device theinvention is not so limited. For example, one or more of the functionsassociated with GSD 110 may be implemented in a plurality of differentnetwork devices, distributed across a peer-to-peer system structure, orthe like, without departing from the scope or spirit of the invention.Thus, as described below in conjunction with FIG. 3, is a network device300 configured to manage a game play using biofeedback measures tomodify a state of the game. However, other configurations are alsoenvisaged.

For example, in another embodiment, the client device 101 may beconfigured to include components from GSD 110 such that client device101 may operate independent of GSD 110. That is, in one embodiment,client device 101 may include game software with biofeedback,biofeedback Application Programming Interfaces (APIs), and the like, andoperate without use of a network connection to GSD 110. Client device101 may therefore, operate as essentially a standalone game device withinterfaces to the biofeedback sensors, and other input/output devicesfor user enjoyment. Therefore, the invention is not constrained orotherwise limited by the configurations shown in the figures.

Although a single client device 101 is illustrated in FIG. 1 having asingle game player and a ‘single set’ of biofeedback sensors 120, otherembodiments are also envisaged. For example, in one embodiment, aplurality of game players, each having their own biofeedback sensorsmight interact and play together a same video game through the sameclient device 101. Thus, multi-player configurations may include suchvariations as multiple game players employing the same or differentclient devices. Therefore, FIG. 1 is not to be construed as beinglimited to a single game player configuration.

Illustrative Client Device

FIG. 2 shows one embodiment of client device 200 that may be included ina system implementing the invention. Client device 200 may include manymore or less components than those shown in FIG. 2. For example, clientdevice 200 may be configured with a reduced set of components for use asa standalone video game device. However, the components shown aresufficient to disclose an illustrative embodiment for practicing thepresent invention. Client device 200 may represent, for example, clientdevice 101 of FIG. 1.

As shown in the figure, client device 200 includes a processing unit(CPU) 222 in communication with a mass memory 230 via a bus 224. Clientdevice 200 also includes a power supply 226, one or more networkinterfaces 250, an audio interface 252 that may be configured to receivean audio input as well as to provide an audio output, a display 254, akeypad 256, an illuminator 258, an input/output interface 260, a hapticinterface 262, and a global positioning systems (GPS) receiver 264.Power supply 226 provides power to client device 200. A rechargeable ornon-rechargeable battery may be used to provide power. The power mayalso be provided by an external power source, such as an AC adapter or apowered docking cradle that supplements and/or recharges a battery.Client device 200 may also include a graphical interface 266 that may beconfigured to receive a graphical input, such as through a camera,scanner, or the like.

Network interface 250 includes circuitry for coupling client device 200to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,global system for mobile communication (GSM), code division multipleaccess (CDMA), time division multiple access (TDMA), user datagramprotocol (UDP), transmission control protocol/Internet protocol(TCP/IP), SMS, general packet radio service (GPRS), WAP, ultra wide band(UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access(WiMax), SIP/RTP, Bluetooth, Wi-Fi, Zigbee, UMTS, HSDPA, WCDMA, WEDGE,or any of a variety of other wired and/or wireless communicationprotocols. Network interface 250 is sometimes known as a transceiver,transceiving device, or network interface card (NIC).

Audio interface 252 is arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 252 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others and/or generate an audio acknowledgementfor some action. Display 254 may be a liquid crystal display (LCD), gasplasma, light emitting diode (LED), or any other type of display usedwith a computing device. Display 254 may also include a touch sensitivescreen arranged to receive input from an object such as a stylus or adigit from a human hand.

Keypad 256 may comprise any input device arranged to receive input froma user. For example, keypad 256 may include a push button numeric dial,or a keyboard. Keypad 256 may also include command buttons that areassociated with selecting and sending images, game play, messagingsessions, or the like. In one embodiment, keypad 256 may include variousbiofeedback sensors arranged to obtain various measures including, butnot limited to pressure readings, response time readings, sweatreadings, or the like.

Illuminator 258 may provide a status indication and/or provide light.Illuminator 258 may remain active for specific periods of time or inresponse to events. For example, when illuminator 258 is active, it maybacklight the buttons on keypad 256 and stay on while the client deviceis powered. Also, illuminator 258 may backlight these buttons in variouspatterns when particular actions are performed, such as dialing anotherclient device. Illuminator 258 may also cause light sources positionedwithin a transparent or translucent case of the client device toilluminate in response to actions.

Client device 200 also comprises input/output interface 260 forcommunicating with external devices, such as a headset, or other inputor output devices, including, but not limited, to joystick, mouse, orthe like. As described above in conjunction with FIG. 1, client device200 may also be configured to communicate with one or more biofeedbacksensors through input/output interface 260. Input/output interface 260can utilize one or more communication technologies, such as USB,infrared, Bluetooth™, or the like. Haptic interface 262 is arranged toprovide tactile feedback to a user of the client device. For example,the haptic interface may be employed to vibrate client device 200 in aparticular way when another user of a computing device is calling.

GPS transceiver 264 can determine the physical coordinates of clientdevice 200 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 264 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, BSS or thelike, to further determine the physical location of client device 200 onthe surface of the Earth. It is understood that under differentconditions, GPS transceiver 264 can determine a physical location withinmillimeters for client device 200; and in other cases, the determinedphysical location may be less precise, such as within a meter orsignificantly greater distances. In one embodiment, however, clientdevice 200 may, through other components, provide other information thatmay be employed to determine a geo physical location of the device,including for example, a MAC address, IP address, or other networkaddress.

Mass memory 230 includes a RAM 232, a ROM 234, and other storage means.Mass memory 230 illustrates another example of computer storage mediafor storage of information such as computer readable instructions, datastructures, program modules or other data. Mass memory 230 stores abasic input/output system (“BIOS”) 240 for controlling low-leveloperation of client device 200. The mass memory also stores an operatingsystem 241 for controlling the operation of client device 200. It willbe appreciated that this component may include a general purposeoperating system such as a version of UNIX, or LINUX™, or a specializedclient communication operating system such as Windows Mobile™, theSymbian® operating system, or even any of a variety of video gameconsole operating systems. The operating system may include, orinterface with a Java virtual machine module that enables control ofhardware components and/or operating system operations via Javaapplication programs.

Memory 230 further includes one or more data storage 244, which can beutilized by client device 200 to store, among other things, applicationsand/or other data. For example, data storage 244 may also be employed tostore information that describes various capabilities of client device200, a device identifier, and the like. The capability information mayfurther be provided to another device based on any of a variety ofevents, including being sent as part of a header during a communication,sent upon request, or the like. Data storage 244 may also be employed tobuffer one or more measures received from a biofeedback sensor.

In one embodiment, data storage 244 may also include cookies, portionsof a computer application, user preferences, game play data, messagingdata, and/or other digital content, and the like. At least a portion ofthe stored data may also be stored on an optional hard disk drive 272,optional portable storage medium 270, or other storage medium (notshown) within client device 200.

Applications 242 may include computer executable instructions which,when executed by client device 200, transmit, receive, and/or otherwiseprocess messages (e.g., SMS, MMS, IMS, IM, email, and/or othermessages), audio, video, and enable telecommunication with another userof another client device. Other examples of application programs includecalendars, browsers, email clients, IM applications, VOIP applications,contact managers, task managers, database programs, word processingprograms, security applications, spreadsheet programs, search programs,and so forth. Applications 242 may further include browser 245,messenger 243, game client 248, and biofeedback device interface (BFI)249.

Messenger 243 may be configured to initiate and manage a messagingsession using any of a variety of messaging communications including,but not limited to email, Short Message Service (SMS), Instant Message(IM), Multimedia Message Service (MMS), Internet relay chat (IRC), mIRC,VOIP, or the like. For example, in one embodiment, messenger 243 may beconfigured as an IM application, such as AOL Instant Messenger, Yahoo!Messenger, .NET Messenger Server, ICQ, or the like. In one embodiment,messenger 243 may be configured to include a mail user agent (MUA) suchas Elm, Pine, MH, Outlook, Eudora, Mac Mail, Mozilla Thunderbird, or thelike. In another embodiment, messenger 243 may be a client applicationthat is configured to integrate and employ a variety of messagingprotocols. Moreover, messenger 243 might be configured to manage aplurality of messaging sessions concurrently, enabling a user tocommunicate with a plurality of different other users in differentmessaging sessions, and/or a same messaging session. As used herein, theterm “active messaging session” refers to a messaging session in which auser may communicate with another user independent of having to restartand/or re-establish the messaging session. Thus, maintaining a messagingsession as active indicates that the messaging session is established,and has not been terminated, or otherwise, placed into a sleep mode, orother inactive mode, whereby messages may not be actively sent and/orreceived.

Browser 245 may include virtually any client application configured toreceive and display graphics, text, multimedia, and the like, employingvirtually any web based language. In one embodiment, the browserapplication is enabled to employ Handheld Device Markup Language (HDML),Wireless Markup Language (WML), WMLScript, JavaScript, StandardGeneralized Markup Language (SMGL), HyperText Markup Language (HTML),eXtensible Markup Language (XML), and the like, to display and send amessage. However, any of a variety of other web based languages may alsobe employed.

Game client 248 represents a game application component that isconfigured to enable a user to select one or more games to play,register for access to the one or more games, and/or launch the one ormore games for online interactive play. In one embodiment, game client248 may establish communications over a network with a network device,such as GSD 110, or the like, to enable registration, purchase, accessto, and/or play of the one or more computer games.

Game client 248 may receive from a user via various user input devices,including, but not limited to those mentioned above, directions tolaunch a computer game. Game client 248 may then enable communicationsof game data between client device 200 and the GSD 110, another clientdevice, or the like.

In one embodiment, game client 248 represents a computer gameapplication; however, game client 248 is not limited to gameapplications, and may also represent virtually any interactive computerapplication, or other interactive digital content. Thus, while describedherein as employing biofeedback measures to modify a state of a videogame play, the invention is not to be construed as being limited tovideo game play, and states of other applications may also be modified.For example, a presentation, tutorial, or the like, may be modifiedbased on biofeedback measures.

Thus, in one embodiment, game engine 248 represents a client componentusable to enable online multi-user game play, and/or single game playeruse. Non-exhaustive, non-limiting examples of such computer gamesinclude but are not limited to Half-Life, Team Fortress, Portal,Counter-Strike, Left 4 Dead, and Day of Defeat developed by ValveCorporation of Bellevue, Wash.

BFI 249 is configured to detect a connection of one or more biofeedbacksensors, and to collect measures received from such sensors. In oneembodiment, BFI 249 may provide information to a remote network device,and/or to game client 248 indicating that a connection with abiofeedback sensor is detected. BFI 249 may further buffer at least someof the received measures. In another embodiment, BFI 249 may select toinstead, provide the received measures to the remote network device,absent buffering, virtually in real-time. In one embodiment, BFI 249 mayconvert the measures into a format and/or protocol usable to communicatethe measures over a network to the remote network device. In anotherembodiment, BFI 249 may select to not communicate the measures over anetwork, such as when client device 200 may be configured as astandalone type of video game console. In one embodiment, BFI 249 mayalso time stamp the received measures such that the measures may bereadily correlated. Further, BFI 249 may provide a sensor sourceidentifier to the measures so that measures may be distinguished basedon its sensor source.

BFI 249 may further perform one or more analysis on the receivedmeasures to determine if a sensor is providing faulty readings, hasbecome disconnected, or the like. Such determinations may be based on acomparison over time of a plurality of received measures for a givensensor to detect changes from an anticipated range of values for areceived measure. For example, if BFI 249 detects that the sensormeasure is a heart rate sensor, and the measures indicate a heart rateof, for example, 2 beats per minute, or even 100 beats per second, thenBFI 249 may determine that the sensor measures are faulty. It should beclear; however, that BFI 249 may employ other range values, and is notconstrained to these example range values. Moreover, BFI 249 may employdifferent range values for different sensors. In one embodiment, BFI 249might provide the determined faulty measures over the network at leastfor a given period of time, under an assumption that the game player istemporarily adjusting the sensor. However, in another embodiment, if thesensor is determined to be faulty beyond the given time period, BFI 249may select to cease transmission of the measures, and/or send a messageto the remote network device.

As noted above, in conjunction with FIG. 1, client device 200 may beconfigured to include components of network device 300 (described belowin conjunction with FIG. 3), including biofeedback APIs, game servercomponents, and the like. In such an embodiment, client device 200 mightoperate essentially as a standalone game console, without communicatingwith network device 300. In such a configuration, client device 200 maybe termed a standalone video game device.

Illustrative Network Device

FIG. 3 shows one embodiment of a network device, according to oneembodiment of the invention. Network device 300 may include many more orfewer components than those shown. The components shown, however, aresufficient to disclose an illustrative embodiment for practicing theinvention. Network device 300 may represent, for example, GSD 110 ofFIG. 1.

Network device 300 includes processing unit 312, video display adapter314, and a mass memory, all in communication with each other via bus322. The mass memory generally includes RAM 316, ROM 332, and one ormore permanent mass storage devices, such as hard disk drive 328, andremovable storage device 326 that may represent a tape drive, opticaldrive, and/or floppy disk drive. The mass memory stores operating system320 for controlling the operation of network device 300. Anygeneral-purpose operating system may be employed. Basic input/outputsystem (“BIOS”) 318 is also provided for controlling the low-leveloperation of network device 300. As illustrated in FIG. 3, networkdevice 300 also can communicate with the Internet, or some othercommunications network, via network interface unit 310, which isconstructed for use with various communication protocols including theTCP/IP protocol, Wi-Fi, Zigbee, WCDMA, HSDPA, Bluetooth, WEDGE, EDGE,UMTS, or the like. Network interface unit 310 is sometimes known as atransceiver, transceiving device, or network interface card (NIC).

The mass memory as described above illustrates another type ofcomputer-readable media, namely computer storage media.Computer-readable storage media may include volatile, nonvolatile,removable, and non-removable media implemented in any method ortechnology for storage of information, such as computer readableinstructions, data structures, program modules, or other data. Examplesof computer-readable storage media include RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by a computing device.

The mass memory also stores program code and data. In one embodiment,the mass memory may include data store 356. Data stores 356 includesvirtually any component that is configured and arranged to store dataincluding, but not limited to game player preferences, game play stateand/or other game play data, messaging data, biofeedback measures, andthe like. Data store 356 also includes virtually any component that isconfigured and arranged to store and manage digital content, such ascomputer applications, video games, and the like. As such, data stores356 may be implemented using a data base, a file, directory, or thelike. At least a portion of the stored data may also be stored on harddisk drive 328, a portable device such as cd-rom/dvd-rom drive 326, oreven on other storage mediums (not shown) within network device 300 orremotely on yet another network device.

One or more applications 350 are loaded into mass memory and run onoperating system 320. Examples of application programs may includetranscoders, schedulers, calendars, database programs, word processingprograms, HTTP programs, customizable user interface programs, IPSecapplications, computer games, encryption programs, security programs,VPN programs, SMS message servers, IM message servers, email servers,account management and so forth. Applications 350 may also include webservices 346, message server 354, game server with biofeedback (GSB)352, and Biofeedback APIs (BAPI) 353.

Web services 346 represent any of a variety of services that areconfigured to provide content over a network to another computingdevice. Thus, web services 346 include for example, a web server,messaging server, a File Transfer Protocol (FTP) server, a databaseserver, a content server, or the like. Web services 346 may provide thecontent over the network using any of a variety of formats, including,but not limited to WAP, HDML, WML, SMGL, HTML, XML, cHTML, xHTML, or thelike.

Message server 354 may include virtually any computing component orcomponents configured and arranged to manage messages from message useragents, and/or other message servers, or to deliver messages to amessage application, one another network device. Message server 354 isnot limited to a particular type of messaging. Thus, message server 354may provide capability for such messaging services, including, but notlimited to email, SMS, MMS, IM, IRC, mIRC, Jabber, VOIP, and/or acombination of one or more messaging services.

GSB 352 is configured to manage delivery and play of a video game usingbiofeedback information obtained from one or more client devices, suchas client device 101 of FIG. 1. Typically, GSB 352 may providecomponents to an application, such as a game application to the clientdevice over a network. In one embodiment, at least one of the componentsprovided is encrypted using any of a variety of encryption mechanisms.For example, in one embodiment of the invention, Crypto++, anopen-source class library of cryptographic techniques, is employed inencrypting or decrypting components of the application. However,virtually any other encryption and decryption mechanism may be used.

GSB 352 may further receive and/or authenticate a request from a clientdevice for access to an application. GSB 352 may provide for purchase ofan application, such as a computer game, enable registration for play ofthe application, and/or enable download access for the application.

GSB 352 may further enable communications between client devicesparticipating in a multi-player application by receiving and/orproviding various data, messages, or the like, between the clientdevices.

GSB 352 may query Biofeedback APIs (BAN) 353 for information about oneor more game player's state or arousal, and/or other information aboutthe game player(s). GSB 352 may then modify a state of the video gameplay based on the received responses to the query. Non-limiting,non-exhaustive examples of queries that GSB 352 might submit to BAPI 353are described below in conjunction with FIG. 6. Non-limiting,non-exhaustive examples of possible ways in which a video game playmight be modified are described below in conjunction with FIGS. 7-8. Inone embodiment, GSB 352 may generally employ processes such as describedbelow in conjunction with FIGS. 5-6 to perform at least some of itsactions.

BAPI 353 is configured to perform various analysis from the receivedbiofeedback measures and to provide responses to various queries fromGSB 352. In one embodiment, BAPI 353 may collect and store receivedbiofeedback measures in data store 356 to enable data analysis to beperformed, auditing over a time period to be performed, historical datato be collected and analyzed, or the like. In one embodiment, BAPI 353may perform at least some analysis upon the received biofeedbackmeasures substantially in real-time. That is, as soon as the measuresare received by BAPI 353, at least some analysis is performed on themeasures.

As noted, BAPI 353 may receive biofeedback measures from a variety ofdifferent biofeedback sensors, including, but not limited to thosedescribed above in conjunction with FIG. 1. In one embodiment, thereceived measures may be identified as a sensor source, such as a heartrate sensor, a galvanic skin sensor, or the like.

BAPI 353, as stated, may perform analysis on the received measures. Forexample, BAPI 353 may receive ‘raw’ biofeedback measures, and determinefrom the measures a heart beat based on the measures. In anotherembodiment, BAPI 353 may employ one or more measures to determine otherphysiological information about an associated game player. For example,BAPI 353 might compute a heart rate variability from heart sensormeasures. Similarly, BAPI 353 might compute a standard deviation ofheart rate activity over a defined time period, determine a trend overtime in a heart rate, and/or determine other heart patterns. BAPI 353may analyze frequency spectrums of heart rate data, including breakingdown beat-to-beat intervals into various frequencies using, for example,Fourier transforms, or similar analysis techniques. BAPI 353 may alsoemploy various measures to determine other physiological informationabout the game player including, but not limited to respiration rate,relaxation level, fight or flight data, or the like.

BAPI 353 might store the results of the analysis for use during asubsequent game play, or determine and employ the results, virtually inreal-time. BAPI 353 may further perform various recalibrationactivities, including, such as a progressive recalibration activity. Inone embodiment, the recalibration activities may be performed on thesensors, and/or to account for physiological changes over time.

Similarly, BAPI 353 may employ historical data based on the biofeedbackmeasures to recognize a particular game player, profiles, or the like,through various mechanisms, including, pattern matching, or the like.BAPI 353 may further recognize when one game player disconnects from thesensors and/or is replaced by another game player, based on suchactivities as missing and/or corrupt biofeedback measures, patternchanges, or the like.

BAPI 353 may also be configured to detect particular patterns,conditions, or the like from analyzing the received biofeedbackmeasures. For example, in one embodiment, BAPI 353 might detect and/oreven predict an onset of motion sickness based, for example, on a causalcoherence between a heart rate, blood pressure, and/or other measures.However, BAPI 353 may further detect other situations that may be of aseverity that warrants sending of an alert message to the video gameplayer, and/or to GSB 352 to cease game play. However, BAPI 353 is notconstrained to these actions, and others may also be performed.

As noted above, BAPI 353 is further configured to make inferences abouta state of arousal, emotional states, or the like, of a game playerbased on analysis of the received biofeedback measures. Such inferencesmay be performed based on the measures as received, and/or based onhistorical data about the game player, and/or other game players. GSB352 may query BAPI 353 for information about one or more game player'sstate or arousal, and/or other information about the game player(s)based in part on the inferences.

In one embodiment, GSB 352 may send a query request for informationabout the game player's state of arousal. In response, BAPI 353 mayprovide a qualitative response, such as “is happy,” “is sad,” “isstressed,” “is lying,” or the like. However, in another embodiment, theresponse may be a quantitative response indicating a level of happiness,such as from zero to ten, or the like. However, the invention is notrestricted to these values or even to this example range, and clearly,other values and/or ranges may be used. For example, a quantitativeresponse indicating a level of happiness could also be a letter grade.

In any event, FIG. 6 illustrates one embodiment of non-exhaustive,non-limiting examples of queries that GSB 352 may send to BAPI 353. Forexample, as illustrated, GSB 352 may send a query seeking to determineif the game player “is frustrated.” Similarly, GSB 352 may send a queryseeking to determine if the game player is “bored,” “relaxed,” “zoning”(indicating that the game player is not focused on the game play), orthe like. GSB 352 could also query whether the game player is“anticipating” some action. Such information may be based, for example,on skin conductance levels, heart rate measures, or the like.

GSB 352 may also send a query seeking specific biofeedback information,such as “determine heart rate trend,” “determine SCL trend” (for skinconductance level), or the like. GSB 352 may further query seekinginformation about the game player's past status, such as “was playerstartled,” or the like.

As illustrated in FIG. 6, GSB 352 may also send query requests toprovide information about the game player as compared to otherinformation. For example, as shown, GSB 352 may query to obtain acomparison between a current state and a previous state of the gameplayer, as well as perform a comparison of the game player to other gameplayers, a baseline, a benchmark, or the like. While FIG. 6 providesnumerous examples of possible queries, it should be apparent that otherqueries may also be performed. Thus, the invention is not constrained tothese examples.

In any event, GSB 352 then employs the results of the queries to modifya state of game play in any of a variety of ways. In one embodiment, andas used herein, a result of a query to GSB 352 may then provide a resultthat may be termed as biofeedback information or a “biocharacteristic.”Using such biocharacteristics obtained from biofeedback of the gameplayer is directed towards providing a more immersive experience of gameplay over traditional game play. For example, the state of the game playmay be modified by enabling avatar mimicry of a player's emotionalstate. For example, if the player is determined to be happy, theplayer's avatar may be modified to appear happy. Similarly, if theplayer is determined to be angry, the game state may be modified topresent to the player a different set of game play experiences than ifthe player is determined to be happy.

Further, in at least one embodiment, the biocharacteristics, such as thestate of arousal of the game player may be used to modify acharacteristic of an input and/or input/output user device. For example,a color of a joystick, a level of resistance on the joystick, or thelike, may be modified as a result of a state of arousal of the gameplayer. Similarly, a color of some other input/output user device mightvary based on a heart beat rate, change levels of intensity and/or colorbased on the heart rate, level of stress, boredom, or otherbiocharacteristic indicating a state of arousal of the game player.

It should be noted that while GSB 352 and BAPI 353 are illustrated asresiding in a network device remote from the client device (such asclient device 101 of FIG. 1), the invention is not so constrained. Thus,in another embodiment GSB 352 and/or BAPI 353 may reside in the clientdevice, a plurality of different client devices, and/or across one ormore different network devices. Similarly, BAPI 353 may reside withinGSB 352, without departing from the scope of the invention.

Generalized Operation

The operation of certain aspects of the invention will now be describedwith respect to FIGS. 4-5 and 7-8. FIG. 4 illustrates a flow chart forone embodiment of a process of employing biofeedback measurements from agame player to modify a game play state in a video game. In oneembodiment, process 400 of FIG. 4 may be implemented with a combinationof GSB 352 and BAPI 353 of FIG. 3.

Process 400 of FIG. 4 begins, after a start block, at decision block402, where a determination is made whether biofeedback sensors areconnected. Such determination may be based on a flag, switch, or thelike received from a client device, a gamer server application, or thelike. In another embodiment, a determination may be made based onreceiving biofeedback measures from one or more biofeedback sensors,where the measures are determined to be within an expected range. Forexample, where measures are received for a heart rate sensor thatappears to indicate background noise measurements, it may be determinedthat the sensor is either faulty and/or otherwise not connected, or thelike. In any event, if it is determined that biofeedback sensors are notconnected for the purpose of modifying a state of a game play,processing flows to block 420; otherwise, processing flows to block 404.

At block 420, other user inputs are received. Such other user inputs mayinclude, but are not limited to joystick, game controller, keyboard,mouse inputs, audio inputs, or the like. Such inputs are typicallyconsidered a result of a voluntary or conscious action on the part ofthe game player, as opposed to biofeedback measure inputs. Processingthen continues to block 422, where the state of game play is modifiedbased on such other user inputs. Processing then flows to decision block416, where a determination is made whether the game play is to continue.If game play is to continue, processing loops back to decision block402; otherwise, processing flows to block 418, where game playterminates. Processing then returns to a calling process to performother actions.

Alternatively, if at decision block 402, biofeedback sensors aredetermined to be connected, processing flows to block 404, wherebiofeedback measures are received from one or more biofeedback sensors.In one embodiment, receiving the biofeedback sensors includes performinga quality analysis upon the measures, time stamping the measures,identifying a biofeedback sensor source, or the like. Moreover,receiving such biofeedback measures may include sending the measuresover a network to a biofeedback API, such as described above. Processingthen flows to block 406, where other user inputs are received, includingvoluntary or conscious user inputs as described in conjunction withblock 420. It should be noted that blocks 406 and 408 may occur in,adifferent order, or even be performed concurrently.

Processing then continues to block 408, which is described in moredetail below in conjunction with FIG. 5. Briefly, however, analysis isperformed on the biofeedback measures to generate historical data,and/or perform other analysis to determine a state of arousal or otherbiocharacteristics of the game player. In one embodiment, block 408 maybe performed substantially in real-time, as the biofeedback measures arereceived.

Processing continues to block 410, where a query may be performedbefore, during, and/or after by the game application (or otherinteractive application). Such queries may include, but are not limitedto those described above in conjunction with FIG. 6.

Continuing next to block 411, the state of game play is modified basedon such other user inputs as joystick inputs, game controller inputs,keyboard inputs, audio inputs, mouse inputs, or the like. Processingthen flows to block 412, based on a result of the query to obtain abiocharacteristic of a game player, a state of the game play may bemodified. Examples of modifying a game play state includes, but are notlimited to modifying a type and/or number of opponents in a game;modifying a pace or tempo of the game; increasing/decreasing a timelimit for a game event; modifying a combat, puzzle, or other challengedegree of difficulty; modifying an availability of supplies, power-upitems, and or other aspects of items in the game; modifying a volumeand/or type of sound, music, and/or other audio feature; modifying acolor, or other aspect of the game, including a background feature ofthe game; modifying lighting, weather effects, and/or otherenvironmental aspects within the game; modify a dialog of variouscharacters within the game including possibly modifying an avatarrepresenting the game player; providing or inhibiting game hints,suggestions, modifying an appearance or function of an application, orthe like. For example, in one embodiment a user interface may bemodified based on various biocharacteristics. Similarly, tutorials,instructions, or the like, may also be modified by skipping, slowingdown/speeding up a rate of presentation, or the like. It should beapparent to one of ordinary skill in the art, that other ways ofmodifying a game state may be employed based on the resultingbiocharacteristics from the query. Processing then continues to decisionblock 416, where a determination is made whether to continue game play,as described above.

FIG. 5 illustrates a flow chart for one embodiment of a process ofperforming an analysis of biofeedback measures from a game player foruse in the video game. Process 500 of FIG. 5 may be implemented, in oneembodiment, within BAPI 353 of FIG. 3.

Process 500 begins, after a start block, at block 502, where biofeedbackmeasures are received. Continuing to block 504, other user inputs, suchas voluntary or conscious user inputs are received. In at least oneembodiment, analysis of the biofeedback measures may employ or becomplemented by information obtained from voluntary or conscious userinputs. For example, where a user is typing into a keyboard a particularcommand, text, or the like, the text or command may be used tocomplement an interpretation of a heart rate variability, or the like.Similarly, flowing to block 506, other game state data may beselectively received and employed to further assist in an analysis ofthe biofeedback measures. For example, such game state data mightindicate that the game is presenting to the game player an extremelydifficult challenge, or the like. The heart rate measures might,however, be determined to be that of a typical adult male at rest.

Flowing to block 508, therefore, a first analysis may be performed onthe received biofeedback measures to determine whether there are missingand/or corrupt data. In one embodiment, such determination mightindicate that a biofeedback sensor is faulty, or that a game player hasmoved the sensor, or the like. In one embodiment, where the measures aredetermined to be corrupt or otherwise faulty for a first time period,but found to be non-corrupt or faulty during a second time period, thenan interpolation might be performed to ‘smooth’ the received measures.In another embodiment, the sensor associated with the corrupt/faultymeasures might be marked or otherwise identified as corrupt. In whichinstance, in one embodiment, the measures from the marked sensor may beignored. In still another embodiment, recent, historically known to begood data may be used to replace data determined to be corrupt/faulty,missing, or the like, to ‘bridge’ a time period during, for example,sensor re-adjustment, and/or other perturbances of the data.

Processing then flows to block 510, where a second analysis is performedon the received biofeedback measures using, in part, the other receiveddata, to determine a state of arousal and/or other biocharacteristics ofthe game player. Using the combination of information during block 510it may be determined that the game player is bored, zoning, or the like.In any event, it should be noted that blocks 502, 504, 506 and 508 mightbe performed in another order, or even concurrently.

A described herein, a variety of mechanisms may be used to infer abiocharacteristic, and/or other physiological characteristics of thegame player, including performing statistical analysis, patternmatching, or the like. In one embodiment, historical information aboutone or more game players may be used to assist in performing theanalysis to infer various biocharacteristics of the game player,including a state of arousal of the game player.

Processing then flows to block 512, where, in one embodiment, at leastsome of the inferences, measures, and/or other data, may be used toupdate a user profile. Processing then flows to block 514, whereselected priority conditions based on the inferences, biofeedbackmeasures, and/or other data may be identified. For example, in oneembodiment, where it might be determined that a game player's measuresare usable to infer that the game player is feeling ill, such conditionmight be identified for further actions. Thus, processing flows next todecision block 516, where a determination is made whether any suchpriority conditions are identified. If so, processing flows to block520, where an alert may be sent to the game player, an administrator, orthe like. In one embodiment, the game play might be terminated.Processing then flows to decision block 518.

If however, no priority condition is identified, processing flows todecision block 518, where a determination is made to continue performinganalysis on the received biofeedback measures. If so, processing loopsback to block 502; otherwise, processing may return to a callingprocess.

There following describes a couple of possible use cases describing useof biofeedback measures to modify a state of a game play. It should benoted, however, that the invention is not constrained to these usecases, and others may also be employed.

As noted above, FIG. 6 illustrates one embodiment of a non-exhaustive,non-limiting example of queries for use in querying a biofeedbackapplication programming interface (API) for biofeedback measures. Itshould be noted that the invention is not limited to these queryexamples illustrated in FIG. 6, and others may also be employed.However, as shown, a variety of different queries may be performed thatinclude, but is not limited to determining a player's arousal leveland/or emotional level. In one embodiment, specific queries regardingarousal might include, is the player “happy,” “sad,” “frustrated,”“energized,” “engaged” (in the game play), “bored,” “relaxed,” or even“zoning.” Queries may also be performed regarding whether the player isdetermined to be anticipating some action, is startled, was startled, orthe like. Similarly, specific biofeedback may be obtained that includes,for example, heart rate trend, an SCL trend, or some other signal trend.In embodiment, a time period may be provided with the query over whichthe trend is to determined.

The queries are not limited to these examples, and other queries mightinclude, comparing information about the player, and/or another player.In one embodiment, an arbitrary query might be generated. For example, aparticular formula, equation, combination of biofeedback measures, orthe like, may be submitted.

FIG. 7 illustrates one embodiment of a non-exhaustive non-limitingexample of using biofeedback measures for use in modifying a game playstate in an arena combat video game.

As illustrated, process 700 of FIG. 7 begins, after a start block, atblock 702, where a computer game that is configured to provide a combatscenario is executed. Execution of the computer game places the playerin a combat arena. That is, in one embodiment, an avatar, or mechanismmay be employed to represent the player within the computer game. Theplayer is employing one or more biofeedback sensors, such as thosedescribed above.

Thus, processing flows to block 704, where a request may be made duringthe computer game to request that the BAPI establish a baseline ofreadings of biofeedback measures for the player. In one embodiment, thebiofeedback measures may include a heart rate baseline, a skinconductance level, or other biofeedback measures that may then beanalyzed to determine a baseline state of arousal or biocharacteristicfor the player.

Processing then proceeds to block 706, where an enemy is introduced intothe arena for combat with the player. In one embodiment, the selectionof the enemy is based on the determined baseline state of arousal. Inone embodiment, the baseline may be used to detect whether this playeris associated with a user profile indicating that the player has playedthis game or a similar game before. Based on the user profile, the enemymay also be selected at a level determined to sufficiently challenge theplayer without boring, or frustrating the player.

Processing moves next to block 708, where the combat is played outbetween the player and the provided game enemy. As the combat is playedout, various biofeedback measures are collected, recorded, and/oranalyzed.

In one embodiment, processing then flows to decision block 710, where adetermination is made whether the combat is resolved. That is, has theplayer or the game enemy won? If the combat is resolved, processing mayflow to decision block 712; otherwise, processing may loop back to block708.

At decision block 712, a determination can be made whether the playerdefeated the game enemy. If so, processing flows to block 714;otherwise, processing flows to block 722.

It should be noted, that in another embodiment, decision block 710 mightbe removed, such that a determination can be made during the samecombat. That is, decision block 712 might be modified, with decisionblock 710 removed, such that a determination is made whether the playeris defeating or winning against the game enemy. In this manner, changesto the game state may dynamically modify a same game combat.

In any event, at block 722, a query may be provided to the BAPI toanalyze the biofeedback measures obtained during the combat of block708. In one embodiment, the analysis may include a comparison of thestate of arousal during block 708 to the state of arousal determinedfrom the baseline for the player from block 704.

Processing then flows to decision block 724, where a determination ismade whether the player had a low state of arousal during the combat.Such determination may be based on whether the difference from thecomparison at block 722 is above a defined threshold value. In anotherembodiment, a statistical analysis may be performed to determine whetherwithin some confidence level, the player is determined to besignificantly aroused statistically. In any event, if the player isdetermined to be aroused, processing flow to block 728, where anotherenemy might be introduced to the game that has a similar level of power,or difficulty as the previous enemy. Processing then flows back to block708.

If, however, the state of arousal is determined not to be statisticallysignificant, or is below some threshold value, then processing flows toblock 726, where a less powerful enemy than the previous enemy isintroduced. Processing then flows back to block 708.

If, however, at decision block 712, it is determined that the player isdefeated or is being defeated, processing flows to block 714, where aquery is performed substantially similar to block 722. Continuing, atdecision block 716, a determination is made whether the player state ofarousal is low, substantially similar to the determination of decisionblock 724. If the state of arousal is low, processing flows to 718;otherwise, processing flows to block 720.

At block 718, a more powerful enemy than the previous enemy isintroduced. Processing then loops back to block 708. At block 720, anenemy having similar power to the previous enemy may be introduced.Processing also then loops back to block 708.

Clearly, where process. 700 is modified to make dynamic modifications ofthe power of the enemy while the same combat is being waged,substitution of the enemy may take several forms, including, forexample, merely enhancing or removing some power from the current enemy;introducing and/or removing additional enemies, or the like.

FIG. 8 illustrates another embodiment of another non-exhaustive non-limiting example of using biofeedback measures for use in modifying agame play state. In process 800 of FIG. 8, the game illustrated is aspace video game. In this example game, the player is challenged toattempt to conserve an amount of oxygen by attempting to control theirconsumption of air. For example, the game may introduce the player to asituation where they are to be rescued in a given time period, such asfive minutes. However, the player's spacesuit contains six minutes worthof oxygen, if consumed at a predefined “regular” rate of consumption ofsay, one unit of oxygen per second. The player is then introduced tovarious situations that may be modified based on the player'sbiofeedback measures. Thus, in one embodiment, the game state could bemodified to make the game more complex or less complex, introducing moreactivities or decreasing a number of activities the player need performbased on the player's biofeedback measures. During the game play, theplayer is further expected to manage their oxygen consumption. Thus, theplayer is challenged to control their air consumption, in oneembodiment, by trying to maintain a reduced level of physiologicalarousal—which may be associated with the consumption of oxygen by thevideo game avatar—while dealing with various stressful tasks within thevideo game, such as combat against an enemy, solving a puzzle, or otherproblem, or the like.

As shown, in this example, then, process 800 begins, after a startblock, at block 804, where various game variables may be set, including,for example, a time for the game, an oxygen level, a consumption rate,and the like. Flowing next to block 806, instructions, or similarinformation, or the like, may be displayed to the player. During displayof the instructions, or the like, various biofeedback measures may bereceived and analyzed to determine a baseline for the player. Forexample, in one embodiment, the biofeedback measures may include a heartrate measure for the player.

Processing continues to block 808, where the BAPI may be queried todetermine an average heart rate for the player over some period of time.As shown in FIG. 8, one period of time is 30 seconds. However, as shouldbe apparent, the game time periods, as well as other parameters, aremerely for illustration, and other values may be used. In any event, theresult of the query may then be used as a baseline heart rate.

Processing next flows to block 812, where the game play may begin.Continuing to block 814, the player is introduced to various game statesof play that may include having the player move, perform combat, playmusic, and/or otherwise repair items, talk to other players, or thelike. Flowing to block 816, during the game play, the game performsadditional query requests to collect additional heart rate measures. Anaverage heart rate may then be determined over some period of time, suchas a most recent ten second of game play. Continuing to block 818, aconsumption rate of oxygen may be further determined based, for example,on a rate at which the player is determined to consume oxygen, based onthe biofeedback measures. In one embodiment, the oxygen consumption maybe derived or otherwise inferred from a ratio of the player's currentheart rate to the average baseline heart rate for the player.

Continuing to block 820, the time for the game play is decremented.Continuing to block 822, an amount of oxygen remaining is determinedbased on the determined consumption rate of the player. Moving todecision block 824, a determination may be made whether there is anymore oxygen remaining. If so, processing flows to decision block 828;otherwise, processing flows to block 826.

At block 826, it is determined that the player's character has run outof oxygen, and therefore has died of suffocation. The game may thenterminate and return to a calling process. Alternatively, at decisionblock 828, a determination is made whether the time remaining in thegame is zero. If so, then the game is determined to be over, and theplayer's character is determined to have survived, at block 830.Processing then returns. However, if there is still more time,processing loops back to block 814 to continue to the game.

It will be understood that each block of the flowchart illustration, andcombinations of blocks in the flowchart illustration, can be implementedby computer program instructions. These program instructions may beprovided to a processor to produce a machine, such that theinstructions, which execute on the processor, create means forimplementing the actions specified in the flowchart block or blocks. Thecomputer program instructions may be executed by a processor to cause aseries of operational steps to be performed by the processor to producea computer implemented process such that the instructions, which executeon the processor to provide steps for implementing the actions specifiedin the flowchart block or blocks.

Accordingly, blocks of the flowchart illustration support combinationsof means for performing the specified actions, combinations of steps forperforming the specified actions and program instruction means forperforming the specified actions. It will also be understood that eachblock of the flowchart illustration, and combinations of blocks in theflowchart illustration, can be implemented by special purposehardware-based systems, which perform the specified actions or steps, orcombinations of special purpose hardware and computer instructions.

As can be seen from the above game examples, biofeedback measures may beused in a variety of ways to modify a state of a game play. However, thevariations are not limited to those described above. For example, invariation to the above games, the biofeedback measures may be used tocontrol an input to the game. For instance, if a large creature ishunting the game player's character, then the player might be expectedto maintain or reduce their stress level to avoid alerting the creatureof their positions. In a similar game, the player might be required todemonstrate sharp physiological arousal to break out of handcuffs orother restraints or break through a locked door to escape a threat.

In another game scenario, little elf characters that give cookies to theuser might only appear when the player is calm, and stay away if theuser is not determined to be calm. A player desiring the cookies (orother reward) must achieve a state of calmness to attract thecharacters. In still another game scenario, in a forest adventure, itmay be sunny with lush, green trees, when the player is determined to bein a particular physiological state of arousal. As the player deviatesfrom that state, the sky might darken, the trees may wither and/orblacken, and/or various colors, music, and/or other sounds may change.Thus, various background aspects within the game may be dynamicallymodified based on the biofeedback measures of the player.

Similarly, based on biofeedback measures from the player, variousnon-player characters may make dialog choices, vary their display or thelike, including commenting directly on the user's inferred state ofarousal or other biocharacteristic.

In still other examples, a user's avatar might show a visible heart,brain, or other bodily aspect, which may be modified based on thebiofeedback measures. For example, the heart might change color to showboredom, anger, happiness, or the like. Similarly, the heart might beatto coincide with the heart rate of the player. In still anotherembodiment, the heart rate of the avatar might be modified to beslightly slower than the heart rate of the player—to attempt to directthe player to become calm. The avatar's facial expressions may also varyas a result of the inferred player's state of arousal, including showinga smile, a frown, anger, or the like.

In addition, a user interface device, screen display or the like, mightbe modified based on the players' inferred state of arousal. Thus, if itdetermined that the player is stressed, the user interface might displaya help feature to guide the player to a solution for a problem in thegame play they are experiencing. There are still a plurality of otherways in which the biofeedback measures may modify a state of game play.Thus, as noted, the invention is not limited to those described above.

However, the above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theinvention. Since many embodiments of the invention can be made withoutdeparting from the spirit and scope of the invention, the inventionresides in the claims hereinafter appended.

The various embodiments described above can be combined to providefurther embodiments. All of the commonly assigned US patent applicationpublications, US patent applications, foreign patents, foreign patentapplications and non-patent publications referred to in thisspecification and/or listed in the Application Data Sheet, including butnot limited to U.S. application Ser. No. 15/369,625 filed on Dec. 5,2016, Ser. No. 12/501,284 filed on Jul. 10, 2009 are incorporated hereinby reference, in their entirety.

1. A video game device, comprising: a nontransitory memory that stores executable instructions; and a processor coupled to the nontransitory memory, in operation, the processor executes the instructions to: provide game play to a video game player via a user interface that provides functionality for a video game; receive, from one or more physical biofeedback sensors, biofeedback measures for the video game player while the video game player is playing the video game; analyze the received biofeedback measures together with historical biofeedback measures to make at least one inference about a physiological condition of the particular video game player; generate a threshold for the physiological condition for the particular video game player based on at least one of historical biofeedback measures for the particular video game player, the current video game play of the particular video game player, or statistics for a plurality of video game players; and dynamically modify or augment the game play of the video game based at least in part on the inferred physiological condition of the particular video game player and the threshold.
 2. The video game device of claim 1 wherein the processor dynamically modifies the displayed user interface.
 3. The video game device of claim 1 wherein the processor dynamically modifies the functionality provided for the video game.
 4. The video game device of claim 1 wherein the processor receives biofeedback measures from at least one camera.
 5. The video game device of claim 1 wherein the processor receives biofeedback measures from at least one camera coupled to a head-mounted device.
 6. The video game device of claim 1 wherein the processor receives a biofeedback measure that includes at least one of heart rate, heart rate variability, heart activity, heart pattern, blood oxygen level, skin conductance level, respiration rate, skin tension, voice stress level, blood pressure, facial expressions, body temperature, pupil dilation, eye movement, gestural motion, Electroencephalography (EEG) or other neural imaging measure, Electromyography (EMG) measure, movement of one or more body parts, or voice stress level.
 7. The video game device of claim 1 wherein the physiological condition inferred comprises a stress level of the particular video game player.
 8. The video game device of claim 1 wherein, to dynamically modify or augment the game play of the video game, the processor at least one of sends an alert message or ceases game play.
 9. The video game device of claim 1 wherein the physiological condition inferred comprises at least one of motion sickness, anxiety, boredom, anger, or frustration.
 10. The video game device of claim 1 wherein the processor uses one or more of the received biofeedback measures to determine that a current player has changed from the particular video game player, and modifies the provided functionality in response to determining that the current player has changed.
 11. The video game device of claim 1 wherein receiving the biofeedback measures includes receiving data for the particular video game player from at least one of eyeglasses, a wrist band, a finger sensor attachment, a sensor within a computer mouse, a sensor within a video game controller device, a microphone, a video camera, or one or more position sensor pads on which the particular video game player stands.
 12. A method operating within a computer device, comprising: providing a current interaction of an interactive application to a user, including displaying, to the user, a user interface with information for the interactive application on one or more displays; receiving, from one or more physical biofeedback sensors, biofeedback measures for the user while the user is interacting with the interactive application; analyzing the received biofeedback measures together with historical biofeedback measures to make at least one inference about a physiological condition of the particular user; generating a threshold for the physiological condition for the particular user based on at least one of historical biofeedback measures for the particular user, the current interaction with the interactive application by the particular user, or statistics for a plurality of users; and dynamically modifying or augmenting the interaction of the interactive application based at least in part on the inferred physiological condition of the particular user and the threshold.
 13. The method of claim 12 wherein receiving biofeedback measures comprise receiving at least one of heart rate, heart rate variability, heart activity, heart pattern, blood oxygen level, skin conductance level, respiration rate, skin tension, voice stress level, blood pressure, facial expressions, body temperature, pupil dilation, eye movement, gestural motion, Electroencephalography (EEG) or other neural imaging measure, Electromyography (EMG) measure, movement of one or more body parts, or voice stress level.
 14. The method of claim 12 wherein analyzing the received biofeedback measures together with historical biofeedback measures to make at least one inference about a physiological condition of the particular user comprises analyzing the received biofeedback measures together with historical biofeedback measures to make at least one inference about a stress level of the user.
 15. The method of claim 12 wherein analyzing the received biofeedback measures together with historical biofeedback measures to make at least one inference about a physiological condition of the particular user comprises analyzing the received biofeedback measures together with historical biofeedback measures to make at least one inference about at least one of motion sickness, anxiety, boredom, anger, or frustration.
 16. The method of claim 12 wherein the interactive application is a video game, and providing of the current interaction of the interactive application comprises providing functionality for the video game to the user.
 17. The method of claim 12 wherein receiving biofeedback measures comprises receiving data for the user from at least one of eyeglasses, a wrist band, a finger sensor attachment, a sensor within a computer mouse, a sensor within a video game controller device, or a microphone.
 18. A video game device, comprising: a nontransitory memory that stores executable instructions; and a processor coupled to the nontransitory memory, in operation, the processor executes the instructions to: receive, from one or more physical biofeedback sensors, biofeedback measures for a video game player while the video game player is playing a video game that provides a user interface that provides functionality for the video game; analyze the received biofeedback measures to make at least one inference about the state of arousal of the video game player; generate a threshold for the state of arousal for the particular video game player based on at least one of: historical biofeedback measures for the video game player, the current video game play of the video game player, or historical biofeedback measures for a plurality of video game players; compare the inferred state of arousal of the video game player to the threshold; and modify the user interface based on the comparison between the inferred state of arousal and the threshold.
 19. The video game device of claim 18 wherein the processor modifies the user interface to dynamically adjust presentation of a game play instruction or a tutorial by skipping, slowing down, or speeding up the presentation of the game play instruction or tutorial responsive to the comparison of the state of arousal of the video game player to the threshold.
 20. The video game device of claim 18 wherein modifying the user interface comprises modifying an output of a display of the user interface based on the comparison between the inferred state of arousal and the threshold.
 21. The video game device of claim 18 wherein the processor receives biofeedback measures from a camera.
 22. The video game device of claim 21 wherein the processor receives a biofeedback measure comprising at least one of heart rate, heart rate variability, respiration rate, voice stress level, facial expression, body temperature, pupil dilation, eye movement, gestural motion, head movement, or body movement.
 23. The video game device of claim 18 wherein the processor receives biofeedback measures from a head worn device that measures a biofeedback measure comprising at least one of heart rate, heart rate variability, heart activity, heart pattern, blood oxygen level, skin conductance level, respiration rate, skin tension, voice stress level, blood pressure, facial expression, body temperature, pupil dilation, eye movement, gestural motion, head movement, body movement, electroencephalography (EEG) or other neural imaging measure, or electromyography (EMG).
 24. The video game device of claim 18 wherein modifying the user interface comprises modifying the user interface to provide an opportunity for the video game player to relax or recover responsive to determining that the state of arousal is above the threshold.
 25. The video game device of claim 18 wherein modifying the user interface comprises modifying the user interface to provide an increased level of excitement for the video game player responsive to determining that the state of arousal is below the threshold.
 26. The video game device of claim 18 wherein the threshold is based on statistical analysis of historical biofeedback measures for the video game player or a plurality of game players.
 27. The video game device of claim 18 wherein the video game is a multi-player video game, and the processor: obtains biofeedback measures from other game players; and further modifies the user interface based on the obtained biofeedback measures from the other game players.
 28. The video game device of claim 18 wherein the processor modifies the user interface to dynamically adjust a difficulty of the video game responsive to the comparison of the state of arousal of the video game player to the threshold.
 29. The video game device of claim 18 wherein the processor modifies the user interface to dynamically adjust an emotional or physiological state of a game character displayed by a display of the user interface responsive to the comparison of the state of arousal of the video game player to the threshold.
 30. The video game device of claim 18 wherein the processor dynamically modifies a characteristic of an user input device operated by the game player responsive to the comparison of the state of arousal of the video game player to the threshold.
 31. The video game device of claim 18 wherein the processor at least one of: modifies a type or number of opponents in the video game, modifies a pace or tempo of the video game, increases or decreases a time limit for a game event of the video game, modifies a degree of difficulty of a challenge of the video game, modifies an availability of supplies or power-up items in the video game, modifies an audio feature of the video game, modifies a video feature of the video game, modifies an environmental aspect within the video game, modifies a dialog of a character within the video game, or provides or inhibits hints or suggestions for the video game.
 32. The video game device of claim 18 wherein the processor dynamically modifies the game state to challenge or entice the game player to maintain a reduced level of arousal. 